Method and apparatus for determining health status

ABSTRACT

Systems and methods for determining a heath status of a patient through an automated interview. One of the systems include one or more computers in one or more locations and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations including: providing, to a user interface of a user device, questions for a user to respond to in an interactive manner, in which each of the questions following the first questions is adaptive based on the user&#39;s response to one or more of the previous questions; capturing motion and appearance of the user in a video sequence while the user is responding to the questions; and analyzing the motion of the user in the video sequence to determine one or more indications of a disease or of a change in a disease progression.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application Ser.No. 62/560,523, filed on Sep. 19, 2017. The disclosure of the priorapplication is considered part of and is incorporated by reference inthe disclosure of this application.

FIELD

This disclosure relates generally to monitoring patient health statusand to the diagnosis and monitoring of disease employing visual andother interactions with live and automated systems, and moreparticularly to visually monitoring and predicting health status byobserving a patient in response to a live or automated interview orother scripted or unscripted interaction to determine response tostimuli, and determine disease state. The disclosure additionally usesfeedback from one or more additional measures of disease in order totune the scripted or unscripted interactions, and allow for more preciseand accurate determinations of disease state.

BACKGROUND

Diagnosis and monitoring of disease traditionally requires subjectivedetermination of disease state by a healthcare professional. Applicationof known disease parameters to a currently observed set of diseasestates from a patient results in a diagnosis of disease. Continuedmonitoring of these disease states allows for monitoring of disease, anddeterminations of progression thereof, over time.

SUMMARY

Existing medical diagnosis and monitoring systems rely on subjectivedeterminations, or upon measurements made in a controlled environment,such as blood draws in a clinic, x-rays and the like. As such, existingsystems fail to describe the use of advanced visual analysis to properlydiagnose and monitor disease, collecting information from acrosspopulations, and determining critical characteristics indicative of suchdiagnoses. These systems similarly fail to take into account accuratedeterminations of disease progression in order to optimize systems forfuture prediction of disease.

The subject matter of the present disclosure aims to address thedrawbacks of the prior system by providing a system and method foranalyzing a video sequence of a user performing one or morepredetermined activity sequences to determine one or more featuresrepresentative of one or more diagnostic attributes.

The contents of U.S. Pat. Nos. 8,781,856, 8,731,961, 8,666,781,9,454,645, and 9,183,601 are incorporated by reference in their entiretyin the disclosure of this application. The subject matter described inthis specification is directed to a system, method and apparatus thatallow for complete control and verification of adherence to a prescribedmedication protocol or machine or apparatus use in a clinical trialsetting, whether in a healthcare provider's care, or whenself-administered in a homecare situation by a patient.

U.S. patent application Ser. No. 15/682,366 filed Aug. 21, 2017 toHanina et al., titled METHOD AND APPARATUS FOR DETERMINING HEALTHSTATUS, the entire contents of this application being incorporatedherein, describes a system for determining health status based uponvisual and other inputs, and a diagnostic system for incorporatingvisual and other sensor data for assisting in making such a diagnosis.The subject matter described in this application builds on these initialapplications and provides one or more interactive systems for engagingpatients to determine disease or progression of one or morecharacteristics of disease, and to utilize feedback from any suchdetermined disease with other measurements of disease to improve anddevelop the interactive system to better perform these functions.

The subject matter described in this application builds on theseprevious applications and additionally provides a system in which apatient may be subject to the conduct of one or more interviews with anautomated interviewer (e.g., a computing device such as a desktop, alaptop, a tablet, a mobile device, or any other device). Theseinterviews are interactive and adaptive, allowing for the interviewer tocollect data from the patient respondent, and adjust the content of theinterview in response to the collected data. Through the use of such anadaptive design, indications of disease progression, changes in diseasecriteria or symptoms prompts additional interview questions or otherinteractions to further flush out details of these progressions ordisease symptoms. The system may be scaled up or down, and can alsoallow for the segmentation of patient risk. That means the system canperform more or fewer steps as necessary. For example, if one particularsymptom is noted to be changing, the system may scale up by takingadditional measurements to further investigate the symptom. For example,if a patient has a higher temperature, the system may automaticallyperform additional tests to determine the source of the temperatureincrease.

The subject matter described herein further provides feedback to improvethe interview and analysis system. Thus, other objective or provenmeasures of disease progression are correlated with interview responses,thus allowing for the improvement of the interview process, and alsoallowing for the focus of the disease progression and determinationalgorithms to more accurately predict disease progression. Indeed,correlation of responses to the interview process and other collecteddata with the outcomes of one or more known measures of disease allowsfor the validation of these analysis techniques, ultimately providing amore objective determination of disease and disease progression, andreducing reliance on, for example, subjective evaluation of response tovalidated assessment scales by trained raters.

Therefore, in accordance with one or more embodiments of the subjectmatter described in this specification, a system and method are providedin which a video sequence of a user performing one or more predeterminedactivity sequences, or performing routine activities, is analyzed todetermine a number of features that may be representative of one or morediagnostic attributes, for example, eye movement, affect, heartrate, andskin tone. Such video sequences and other relevant collected data may beperformed in response to a predetermined test sequence, or may beperformed in response to an interactive interview presented to thepatient, and performed by a live human interviewer, or an automatedinterview system. Once such features are recognized and tracked, asubsequent determination may be made to determine a subset orcombination of these features that are indicative of diagnosis ormonitoring of disease, and may be analyzed over time to determinechanges in a particular disease progression. Such analysis may also bedetermined in accordance with a longitudinal analysis across one or moredisease states.

Images or video sequences associated with a remote interview may becaptured using an image capture device, for example, a dedicated cameraor a camera embedded in a mobile device (e.g., a smartphone, a tablet, alaptop, or a smartwatch). In some embodiments, the image capture deviceis a single, stereo, or depth camera. In some embodiments, the imagecapture device includes additional sensors such as audio, motion, rangeor other sensors. Analysis processing may employ any methods, such ascomputer vision analysis, neural networks, deep learning, machinelearning, or the like. Processing may be provided by a processorembedded within a camera, mobile device, or dedicated computing system,either local or remote, such as in a cloud-based system. Datatransmission takes place over a cellular, Wi-Fi enabled or otherwireless or wired communication system.

Various embodiments of the subject matter described in this applicationmay include the use of 3D mesh tracking (for providing higher resolutiontracking of any face or other body part movement, super high resolutionaction units (for monitoring detailed movement of the user), super highframe rate (to allow for higher resolution viewing and analysis ofmovement of the user, and video magnification (to allow for intensefocus on the most important portions of one or more input images) aspotential inputs. As will be further described below, the systems tracksand assesses patients over time based on a patient's current level ofdisease, or other variable. So system calibrates and identifiesvariances based on individual patient, and applies appropriatemonitoring sequences based upon progression of the patient along one ormore measurable variables.

The system also determines tone, type of questioning and is adaptiveover time. The system does not just ask one validated scale but ratheradjusts to patients responses, and may determine whether further,additional, or varied assessments should be applied to the patient. Suchadjustments may include frequency, tone, intensity and interventiontypes, thus allowing for a varied experience by the patient. The systemmay further simulate nurse or other healthcare provider interactions.The system may therefore demonstrate empathy if patient is sick, and isadaptive to patient needs. The system is therefore interactive. Thesystem is able to train patients to properly perform desired actions.Further, the system effectively monitors patients in either active orpassive manners. In addition, the system intervenes with patients whendetermined to be appropriate, and is empathetic based on risk of thepatient, and the current disease state of the patient. The systemtherefore establishes a bond with the patient, but also prompts theuser, and is able to collect desired information from the user.

Based on a determined disease situation of the patient, and risk ofdisease progression, the system may capture some information morefrequently and/or collect some other information less frequently. Suchtiming may be based upon recommendations for collection of information,based on a link to illness, or based upon an interactive multi-variabledetermination of best interval for administration. The system is furtherable to self-learn interactions based on monitoring of live nurses andcare providers, as will be described in greater detail below. Byobserving actual human interaction, the system is able to monitor how tointeract when performing particular tasks, or administering particulartest sequences, and is able to simulate such interactions. Furthermore,if such interactions have a positive influence on disease or the healthof the patient, such interactions may be employed in the future for anysimilar interactions. This information is provided to a learning engineto ensure empathy of the system, to support the system's ability toinstruct patients to perform particular desired actions, and its abilityto coordinate care.

Based upon the current state of a patient (i.e. demographic informationcurrent and historical disease progression, etc.), an inventive learningengine may also determine what types of assessments provide the mostaccurate results. Tone, speed, content of questions may be modifiedbased on observation of human raters, assessors or care providers, andintegration of these techniques when it is determined that suchactivities provide a positive response from patients. Thus, the systemis able to listen to videoconference interviews, telephone interviews,or even in-person assessments to determine the most effectivetechniques, and to further determine techniques that provide the mostpositive response, based upon disease condition, or other categorizinginformation. Finally, collected data may be re-analyzed in response tonewly-determined interaction techniques to further provide input toadjust actions of the system towards patients.

Still other objects and advantages of the subject matter describedherein will in part be obvious and will in part be apparent from thespecification and drawings.

The subject matter described herein accordingly comprises the severalsteps and the relation of one or more of such steps with respect to eachof the others, and the apparatus embodying features of construction,combinations of elements and arrangement of parts that are adapted toaffect such steps, all as exemplified in the following detaileddisclosure, and the scope of the subject matter described herein will beindicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the subject matter described inthis application, reference is made to the following description andaccompanying drawings, in which:

FIG. 1 is a block diagram depicting the details of an exemplary hardwareconfiguration for implementing the system and method described in thisapplication;

FIG. 2 is a block diagram depicting additional details of an exemplaryhardware configuration for implementing the system and method describedin this application;

FIG. 3 is a flowchart diagram depicting additional details of anexemplary implementation of monitoring to determine disease progression;

FIG. 4 is a timeline depicting a relationship between assessments; and

FIG. 5 is a flowchart diagram depicting an embodiment of the subjectmatter described in this application.

DETAILED DESCRIPTION

In accordance with an embodiment of the present disclosure, a visualmotion capture device, camera or the like is used to capture motioninformation related to the motion of a user, such as a medical patient,while performing one or more predetermined functions. Thesepredetermined functions may be part of a script through which the usermay be guided, may be a movement, speech, or other actions that may bereviewed while the user is performing one or more other tasks, or may bemovement, speech or other action in response to an interview performedwith the user. Such an interview may comprise a live interview with theuser that is recorded and sensed by the recording and sensor devices, ormay comprise an automated interview in which the user interfaces with amobile or other device, the device simulating the actions of a humaninterviewer. The automated interview further includes one or moreinstances of branching logic, so that a measured response to a firstquestion (including one or more of analysis of a visual, audio, orselection response) will result in a determination of a next question orset of questions to be asked. Automated interview questions may be askedover time, and across a number of patients, and therefore providing abaseline of the patient over time. Subsequent measurements are then ableto determine changes in the patient activity over time. By furtheraccumulating assessments over a number of patients over time, a generalaverage baseline across all patients may similarly be provided. Finally,by segregating the accumulated patient information by demographics,disease state, or the like, baseline averages may be provided for eachsuch segment, allowing for the determination of variances thereofindicative of changes in disease progression.

In U.S. patent application Ser. No. 13/189,518, filed Jul. 24, 2011 toHanina et al., titled METHOD AND APPARATUS FOR MONITORING MEDICATIONADHERENCE, the entire contents thereof being incorporated herein, thepresent disclosure describes a system and method for providing a statemachine that utilizes audio/video information to offer a populationhealth tool to manage any number of patients, understand their behavior,and communicate and intervene when necessary or desirable. The system asset forth in the '518 application further employs machine learning toidentify one or more trends and make automated judgments about patientstates, as well as an ability to learn and highlight outliers or at riskpopulations. Thus, based upon captured information, patients may beplaced into states that may aid in predicting those patients at risk forfuture hospitalizations, for example, or other types of situations wherea varied intervention strategy may be beneficial. When considering largepatient populations, such automated monitoring and categorization allowsfor monitoring of such patients, allowing managers to direct theirattention to patients who might best benefit from such attention, andallowing the system to provide automated intervention when determined tobe appropriate. This level of automated, intelligent intervention allowsfor effective management of patient behavior and medication adherence.Existing systems fail to capture patient behavior. Various embodimentsof the inventive solution, including the described state machine,provide data relevant to medication adherence and other medicaltreatments as opposed to entire patient history, such as in an existingelectronic medical record. Furthermore, the system acts as a videorepository, recording administration by patients, and thus allowing forfuture review of such administration sequences by a manager or otherhealthcare professional if appropriate. Thus, upon determination by thesystem in a manner noted above, patients in one or more predeterminedstates may be indicated for such manual review by the manager or otherhealthcare provider. Finally, the system may be applicable not only toadherence information, but to any patient action or healthcare relatedtreatment to which monitoring may be applicable.

The present disclosure additionally expands on these ideas, by usingresults of video, audio or other analysis techniques, to monitor useractivities in response to a remote, automated interview process, forexample, as place the user into a particular state in response to one ormore of such analyzed responses, and additionally based upon one or moremedically significant inputs (i.e., medical data such as a healthstatus, disease state/condition, age, etc.) or a prior state of the useras related to the observation in question. Thus, upon placement of theuser in a particular state, response to a particular interview questionmay result in the selection of a predetermined branching logic to movethe automated interview in a particular direction, this resulting in aset of context and status based questions to be asked of the user.

In response to one or more actions captured and analyzed of the user inresponse to one or more of the posed automated interview questions, andadditionally in accordance with the current state of the user, andadditionally based upon one or more additionally collected pieces ofinformation, a risk of patient behavior may be determined. Such risk maybe associated with the user progressing along a particular disease, arisk of re-hospitalization, a risk of non-adherence to medication, orthe like. Such risk may also be considered placing the user in aparticular state that is defined by this higher risk profile. Inresponse to this escalated risk profile, one or more desired responsesmay be implemented, such as a change in medication prescription, changein therapy, notification of a healthcare provider or caregiver, or achange in any other desired interaction with the user.

Information Capture and Analysis System

FIG. 1 illustrates an information capture and analysis system thatincludes a remote information capture apparatus 1000, a communicationsystem 2000, and a remote data and computing device 3000. Theinformation capture and analysis system is adapted to allow for thecapture and processing of information in order to implement the systemand method in accordance with the present disclosure. The informationcapture apparatus 1000 communicates with a remote data and computingdevice 3000 via a communication system 2000 (e.g., the Internet, Wi-Fi,LAN, WAN, Bluetooth, or other communication system). Via thecommunication system 2000, information captured by apparatus 1000 may betransmitted to remote data and computing device 3000, and analysisinformation or other instructions may be provided from remote data andcomputing device 3000 to apparatus 1000. It is further contemplated thata plurality of such information capture apparatuses 1000 may becoordinated to monitor a larger space than a space that can be coveredby a single such apparatus. Thus, the apparatuses can be made aware ofthe presence of the other apparatuses, and may operate by transmittingall information to one of the apparatuses 1000, or these apparatuses mayeach independently communicate with remote data and computing location,which is adapted to piece together the various information received fromthe plurality of devices 1000.

FIG. 2 shows a more detailed view of an example embodiment of remoteinformation capture apparatus 1000 and remote data and computing device3000 of the information capture and analysis system of FIG. 1. As shownin FIG. 2, the apparatus 1000 comprises an information capture device1110 for capturing video and audio data as desired. A motion detector1115 or other appropriate trigger device may be provided associated withcapture device 1110 to allow for the initiation and completion of datacapture. Information capture device 1110 may comprise a visual datacapture device, such as a visual camera, or may be provided with aninfrared, night vision, or other appropriate information capture device.A storage location 1120 is further provided for storing capturedinformation, and a processor 1130 is provided to control such captureand storage, as well as other functions associated with the operation ofremote information capture apparatus 1000. An analysis module 1135 isprovided in accordance with processor 1130 to perform a portion ofanalysis of any captured information at the remote information captureapparatus 1000. Apparatus 1000 is further provided with a display 1140,and a data transmission and receipt system 1150 and 1160 for displayinginformation, and for communicating with remote data and computing device3000. In some embodiments, display 1140 may be used, along with an audiospeaker, if desired, to provide one or more interview questions inaccordance with an automated interview process. In such a situation,information capture device 1110 would then capture video and audioinformation provided by a user in response to the automated interviewprocess.

The remote data and computing device 3000 comprises system managementfunctions 3030, and a transmission and reception system 3050 and 3060for communicating with apparatus 1000. Transmission and reception system3050 and 3060 may further comprise various GPS modules so that alocation of the device can be determined at any time, and may furtherallow for a message to be sent to one or more individual apparatuses,broadcast to all apparatuses in a particular trial, or being used foradministration of a particular prescription regimen, of broadcast to allavailable apparatuses.

In accordance with an embodiment of the disclosure, apparatus 1000 isadapted to be part of a system that monitors user (patient) diseasecharacteristics by way of passive monitoring, or actively requestingthat the user be interviewed by the system, responses to the interviewquestions, as well as other movement comments or the like observedduring conduct of the interview. Users of apparatus 1000 in accordancewith this system give administrators a tangible and concrete manner inwhich to review activities and collected information. Apparatus 1000 ofthe disclosure is adapted to receive instructions, interview questions,or the like for patients from remote data and computing device 3000 andprovide these instructions to patients or conduct an automated interviewwith the patients. Such instructions may comprise written, audio oraudio/video instructions for guiding a user to perform one or moreactivities, such as performing a sequence of actions to test aparticular action of the user, or whether a user is adhering to aprescribed medication protocol. The video instructions can be providedeither by a real person or by an animated cartoon character (avatar), orthe like.

The system, in accordance with an embodiment of the disclosure, is alsoapplicable to monitoring of patient activities when being requested toperform particular actions in response to a remote, automated interviewprocess, such as when performing such actions in order to simulate apredetermined measurement scale, such as may be used when assessingstatus of a particular disease. Such scaled may comprise one or morevalidated scales, typically administered by a healthcare provider, andcorrelated to provide insight to status of a disease. The conduct of aremote, automated interview will allow for the evaluation of a patientin relation to the particular disease without the need for a human toadminister such a validated scale. Therefore, in accordance with anembodiment of the present disclosure, a method and system may beprovided for analyzing captured patient motion data in near real time toprovide feedback to the user (that is, the analysis is performed quicklyenough to be able to provide feedback to the user while the system isstill in use), to determine a number of times a participant performssome action that is, for example, considered suspicious, or to determineone or more elements of diagnosing or monitoring disease.

In accordance with a further embodiment of the present disclosure, thevisual capture device 1110 may be used to capture visual informationrelated to one or more users. Instructions about how a patient should beinterviewed, proximity to camera, volume of response, etc. may also beprovided. The operator or user may further be provided with the optionto blur or make the screen opaque during interview process andcollection of video, thus protecting the identity of a user capturedtherein.

Any standard camera or image capture device may be employed, includingbut not limited to a camera on a mobile phone, tablet, other computingdevice, standalone camera, or any other image acquisition apparatus thatis able to record one or more (video) images of a subject. Audio andother characteristics may similarly be recorded along with the video, inaccordance with the use of other appropriate sensor devices. In apreferred embodiment of the disclosure, the subject may comprise a faceof a human, but may comprise any other desirable subject, including oneor more other body parts of the user, or other object. Analysis of theserecorded images may be performed currently, or the images may be storedfor future analysis. Storage of such images may be performed local tothe image capture apparatus, at a remote location, such as a dedicatedstorage location, or a cloud based storage system. Performance ofautomated interviews typically require near real time analysis ofincoming data so that any branching logic may be applied, and theinterview may continue. Of course, even after being analyzed locally,further remote analysis of the collected data may be performed.

Additionally, visual representations of a user can be further used todetermine a status of the user in response to any request for activityin accordance with, for example, an automated interview. The system mayperform a remote physiological exam based on touching the user's nose orholding up 3 fingers, for example. Visual determination of one or moreparameters, such as motion, (body motion or camera motion with respectto still object), eye motion, skin tone, emotions, heart rate, breathingpatterns (measured by video/depth image analysis or other devices),blood pressure, body mass, GPS location, proximity, or othermeasurements (such as non-visual measurements) that may be provided inaccordance with one or more incorporated or coupled sensor, may bemeasured visually or otherwise, at once or over time, to determinechanges in one more of such parameters in order to identify changes inthe health of the user. In accordance with the conduct of an automatedinterview process, such parameters may be measured in response to one ormore questions or request in the interview. Analysis of such responsesmay further allow for branching logic to be applied, and to guide thefurther direction of the interview. In accordance with an embodiment ofthe present disclosure, by way of example, display 1140 displays one ormore bits of information to a user, such as a request in accordance witha predetermined interview. Such information may comprise a specificvideo sequence designed to test the reaction of the user, or maycomprise interactive or other instructions to the user to perform apredetermined activity. Information capture apparatus capturesinformation monitoring the user upon viewing of the displayedinformation, and performing one or more activities, or otherwiseresponding, in response thereto. Other devices for capturing informationin response to presented visual, tactile, auditory, olfactory, gustatoryor other stimuli may include diverse sensors, such a glucose meters,blood pressure cuffs, radar systems, visual capture devices,thermometers, accelerometers (measuring the shake of the hand of a user,for example), or the like. One or more of such measure parameters may beused to identify particular characteristics of one or more diseasestates. In such a manner, while monitoring adherence or otheractivities, or when performing actions in response to a presented testscript or questions in an automated remote interview, such parametersmay be measured, and reported back to one or more healthcareprofessionals, one or more care providers, other individuals, or may becollected in order to analyze automatically, perhaps over time, todiagnose and monitor disease. Thus, these parameters may be measuredover time without reference to adherence, allowing for diagnosis ofdisease, measurement of progression of disease once diagnosed, ormeasurement of various health indicators to gauge the overall health ofan individual.

Furthermore, a database or other repository of such measurements may becollected over time and over users at remote data and computing device3000. Such database may be characterized by disease state or otherdemographic characteristics. In this manner future measurements of oneor more users may be compared against such a database, and allow fordiagnosis of one or more diseases, or changes in these characteristicswhen monitoring these diseases. Scales of responses to interviewquestions may also be stored in such a database, and allow for thedetermination of progression of disease, as well as correlation ofresponses to an automated interview with responses to questions duringlive administration of a validated scale testing for status orprogression of disease.

The system may step up or step down levels of assessment based on riskprofiles, so that users determined to be of a higher risk may receivemore scrutiny. The system may also perform testing in accordance withpassive monitoring in background based on visual information as theAiCure platform is used in normal functioning, as well as performingactive assessments. In addition, if necessary, the system can go througha decision tree to offer tele-psychiatry when risk levels are high, orperform a simple check in when risk is low. If it is determined that auser has run out of medication (through calculation of adherence overtime, or otherwise) automated delivery of medication may be provided, orif the patient needs to be remotely titrated, or changed to a differenttherapy. Optimization of the process will be achieved based on link tohealth outcomes (hospitalization events) captured from EMR or other datasource, thus reinforcing actions and assessments that turn out to be themost accurate and helpful.

Furthermore, expected progression of such parameters or responses toquestions over time may be determined for a population as a whole, orfor various portions of a population, defined by demographics, diseasestate or the like. So, by way of example, it may be possible todetermine expected progression of one or more visual characteristics,such as weight gain, of a female aged 40-60 suffering from diabetes, orto determine expected changes in response to a visual presentation of ascript to be followed by a user. Progression of a visually measurableresponse may similarly be determined and correlated with progression ofdisease or other characteristic. Of course, other sub-populations mayalso be determined from such a database. Validated scales may further beemployed to confirm progression of disease.

In yet a further embodiment of the disclosure, the determination ofwhether a particular user has modified disease characteristics may bedetermined in accordance with one or more unsupervised learning systems,such as a neural network or the like. In such a manner, the database ofcollected images may be employed to train such a system, identifying oneor more characteristics from the training images that may be used toidentify similar characteristics in future images. Additionalinformation collected from one or more external sensors, such asaccelerometers, voice recorders, or the like associated with the cameradevice, or one or more external medical devices, such as glucose meters,heartrate meters, or other measurement devices, or any other may befurther included in the unsupervised learning system to additionallycategorize images. This collected information may be used to calibratethe system during a learning phase, and may be subsequently removedduring an operation phase. Combination of one or more of these readingswith visual information may further allow for the determination ofadditional changes in status of a patient or user.

By way of example, pulse oximeters, heartrate monitors and the like maybe employed with collected video information to allow for more precisedeterminations, in either an active or passive mode. Additionally, micromovements associated with movement of a mobile device or the like mayalso be employed. Micro eye movements, gaze tracking, analysis ofexpression, or any other micro gestures, micro movements, or otherrecognizable conditions and the like may further be employed. The AiCuresystem may monitor performance of such actions, and where the user isunable to perform such functions, may instruct the user on how toproperly perform these actions. These additional measured features maybe further employed to identify changes in characteristics along anumber of alternative dimensions in such an unsupervised or supervisedlearning system, ultimately diagnosing or monitoring disease. Analysisof the accumulated information may allow for identification of one ormore common characteristics among or between various disease states,demographic states, or other common identifying characteristic.

Longitudinal analysis of such data and changes in visual and othercharacteristics over time may be further correlated to negative healthoutcomes such as hospitalization events or death, and may give rise torelationships that can then act as the basis to trigger interventions inadvance of a negative health outcome occurring. Through such monitoring,early warning signs may be extracted from visual images of users in amanner not previously possible. Thus, any number of visual analysistechniques may be employed to generate a video asset base by therapeuticarea over time, thus allowing for the use of such assets to evaluate thehealth of users in the future including similar characteristics, andresiding in similar therapeutic areas.

In accordance with one or more embodiments of the disclosure, it isanticipated that the use of one or more sections of the electromagneticspectrum will allow for an in-depth analysis of facial or other visibleuser features. For example, as will be described below, rather thansimply noting external facial features, use of the disclosure allows forthe determination of the location of various blood vessels under theskin of a user in the field of view of a camera. Such analysis can beextended to any body part, and can be combined with any sequence ofuser-performed self-administration steps, or can be implemented with acare provider instructing actions to be taken. Over time, differencesdetermined in the various images provide information about theperformance of the user, and may further indicate changes in disease,physical ability, or the like. Such changes, for example, may be morevisible under near-infrared light, or other wavelength of energy, thusresulting in additional information being extracted based upon the useof multiple types of light, energy, or other data extraction mechanisms.

The system may therefore learn various correlations between one or moreobserved features, and health status, health outcomes, diseaseprogression, symptom progression, or one or more changes in overallhealth. By analyzing and correlating these changes in features andultimate health status, the system provides a mechanism for determiningyet unknown relationships between measurable quantities and the healthof an individual. Once established, these relationships can then be usedto predict future medical situations. By way of example, one or moresub-segments of the population may be targeted for observation. If suchpopulation is a post-stroke population, it is known that rapid weightgain may be a symptom of failure to take proper medication, or maypredict a more urgent medical situation. In accordance with anembodiment of the disclosure, daily images of an individual may allowfor a determination of such rapid weight gain without the use of abody-weight scale. In such a situation, a healthcare provider may beimmediately notified to follow up with the individual. Whilevisual-spectrum features may be used to determine weight gain,determinations of changes in pulse, blood pressure or other measurementsmay rely on the above-mentioned other areas of the electromagneticspectrum, audio pulses, or any other type of desirable sensor, whetheralone or in concert with a visual analysis.

In accordance with alternative embodiments of the disclosure,accumulated images of one or more users, associated sensor datainformation, visually extracted information, and one or more additionalinputs may be incorporated into a comprehensive database. Analysis ofthe accumulated information may allow for identification of one or morecommon characteristics among or between various disease states,demographic states, or other common identifying characteristic. Humaninteractions may be further analyzed, and provided as further input intothe system to determine the effectiveness of such interactions, and toallow the system to simulate human interactions with the user.

Such monitoring may take place in an active or passive monitoringsituation, and may be provided on a mobile platform, and fixed computersystem, or any continuous monitoring system.

FIG. 3 is a flow diagram of an example process for interviewing a user,monitoring the user during the interview to collect information aboutthe user, and analyzing the collected information to determine one ormore diagnostic attributes indicative of a disease or a change in adisease progression. The process is performed by a system of one or morecomputers located in one or more locations. For example, the informationcapture and analysis system of FIG. 1, appropriately programmed, canperform the process.

As shown in FIG. 3, in an active monitoring situation as part of anautomated interview, the system automatically asks the user a sequenceof questions in an interactive manner. Each of the question in thesequence following the first question is adaptive based on one or moreof the responses of the user to the previous questions in the sequence.That is, the system automatically analyzes the content of the user'sresponse to one or more previous questions in the sequence and thenautomatically generates the next question based on the content ofresponses. For example, users suffering from a particular disease mighthave a particular set of initial questions presented to them. Knowledgeof the disease may be used to aid in the interpretation of responses tothe questions. Responses to one or more of the questions in a particularmanner may indicate a progression of disease in a direction thatwarrants follow up questions. A user suffering from Schizophrenia may beasked a set of questions to determine a general level of the disease. Ifanswers to these questions, either actual response words or any observedactions while answering, indicate, for example, that negative symptomsare increasing (i.e. the user has a less animated response to thequestions), additional questions may be presented in order to elicitadditional information from the user to further flush out changes insymptoms of the disease. In another example, if someone is answeringquestions related to likelihood of suicide, answering questions aparticular way may result in further follow up questions to determine amore precise risk of danger, and the need for help. It is the use ofhigh level branching logic that allows the system to present aninteraction that is able to extract the most critical details in anobjective and scientific manner. Based on the answers of the user tothese questions, the system asks the user to perform a particular set ofactions, for example, actions for completing a medical test. The systemmay instruct the user to perform these actions on a mobile or otherlocal device (e.g., the remote information capture apparatus 1000 ofFIGS. 1 and 2) at step 1710. The system may display on a local devicedisplay (for example, the display 1140 of FIG. 3) one or moreinstructions to the user in accordance with a predetermined interview,and then captures (visual, audio, etc.) information related to theperformance of the actions by the user at step 1720. Thus, if the useris to perform an eye movement test in response to one or morepredetermined interview questions, on one embodiment, the system mayinstruct the user to watch an object such as a marker on a display. Thesystem may display the marker on the display. The system can move themarker in a predetermined sequence around the display while the user islooking at the marker. This operation allows the system to measures theuser's ability to maintain focus on a single item moving through a fieldof view. The system may monitor an eye movement (“gaze tracking”) step1730 to determine disease, or if the monitoring is performed over timeby the system, the system can determine progression of disease as shownat step 1740. For example, the system determines disease or progressionof disease based on the ability for a user to follow a moving object, orfocus on a particular portion of a display screen or object. Morespecifically, the system may determine that slower tracking of a movingobject by the gaze of the user is an indication of progression of aparticular disease, such as symptoms of schizophrenia or other disease.Similarly, if gaze tracking is improved, the system may determine that amedication being administered is working to reverse disease progression.Of course, other questions, requests for actions, and observations maybe employed as part of the automated interview process.

In an alternative embodiment, the system may ask the user focus on aparticular marker on the display at step 1710. The system then provideson the display a second marker. The system measures the ability for theuser to continue to focus on the initial marker at step 1730. The systemthus can measure the user's ability to maintain focus, and how easilythe user is distracted. Again, by monitoring the user over time, thesystem can determine progression of disease at step 1740.

In another alternative embodiment, the system may ask the user to hold aphone camera or other hand-held camera to focus on one or more staticobjects. When doing so, the system can use the relative motion of thestill objects in the video (relative to the phone) to measure patienthand stability or tremor. In a manner similar to changes in gazetracking noted above, the system can determine that the increases intremor is an indication of a worsening condition in particular diseases,such as in Parkinson's or Alzheimer's disease. By measuring the relativetremor over time, the system can monitor symptom and diseaseprogression. Similarly, if the system determines that tremors arereduced, the system can infer that the medication that the patient hastaken is effective. By tracking an improvement of symptoms, the systemmay also obtain an evidence that a user is in fact properlyadministering the user's medication.

In a further embodiment of the disclosure, one or more correlationsbetween responses provided by a patient in response to a remotelyadministered automated interview and an expertly administered validatedassessment are determined. A validated assessment may be administered bya healthcare provider, and includes one or more questions or otheractivities that are predetermined to indicate level of disease, therebyproviding the ability to compare the health status of a particularpatient to that of a population as a whole. Such scales may include, forexample, SMWQ/Study Medication Withdrawal Questionnaire for use toevaluate study medication withdrawal symptoms in therapeutic areasincluding mental disorders and chemically-induced disorders. A scoreprovide after administration of this scale provides a score indicativeof the level of medication withdrawal symptoms. Other scales allow forassessment of symptoms or disease progression for any number ofdiseases.

Such a comparison between the known responses to the validated scale,and the automated scale described in accordance with this disclosure maybe made based upon analysis of a large number of interviews performed incomparison to known responses included in the validated scales, and mayemploy any of the structured or unstructured learning techniques notedabove. As part of the noted correlation determination, to the system canevaluate scores of both expertly administered scores, and automaticallyadministered interviews, using both visual and prosodic features, andpassive sensor data (in the automated situation). In a controlledsetting, such as a clinical trial, one may attempt to assign patients toone of two groups indicative of those taking a study drug, and one groupin a placebo control group. Determination of an automated interviewsystem that is able to consistently determine proper groups (as those inthe placebo group should not see any benefit), would thus become anobjective system for determining progression or regression of disease.The system may further comprise a learning engine for empathy, content,accuracy of assessment. The system therefore performs side by side to ahuman rater or care provider to capture interactions and learn how tosimulate the interactions provided by the humans. In other words, apatient can first be evaluated by a doctor administering a validatedscale, and then using the system provided in accordance with thisdisclosure. Results can be compared over time, and correlations betweenthe validated scale and the automated system can be determined in orderto confirm that accuracy and applicability of the automated system. Thesystem is then able to proactively create a decision branch to makeassessments that have the best outputs.

In addition to collecting information in an automated fashion relatedconduct of an automated interview, the system may employ one or morepassive sensors in order to collect further information that may becorrelated with interview responses, and further support determinationof disease progression. Such passive data may include one or more of thefollowing sensors or type of data collected: Speech duration, Geospatialactivity, Kinesthetic activity, Sleep duration and quality data, Number,frequency and duration of phone calls, prosodic features, or other phonemeasured details, App usage (social, engagement, and entertainment), andone or more measurements of the ambient environment, and video recordingof a daily activity like brushing teeth, combing hair, and/or takingmedicine etc.

Referring next to FIG. 4, a mechanism for evaluating one or moreautomated interview systems in accordance with an embodiment of thesubject matter described in this specification is shown. For apredetermined period of time (trial start to trial end), one or morepatients are evaluated upon visits to a healthcare provider (assessmentsY), and is also interviewed by an automated interview system(measurements U), while a recorded version of responses to the automatedinterview may be scored by an expert (assessments W) at timescorresponding to the healthcare provider visits. Thus, comparisons maybe made to correlate the assessments Y from the healthcare provider andassessments W based on the recorded responses of the user to theautomated interview (hereafter referred to as “automated responses”).Furthermore, a supervised learning system can be trained toautomatically infer estimates of assessments Y, from measurements U andpossibly taking advantage of assessments W either only during trainingor during training and testing. Additionally, between healthcare officevisits, the automated interview measurements X, and expert scoring ofrecordings of these interviews (assessments Z) may similarly becorrelated to determine further correspondence between the assessments.Measurements X can be taken more frequently and in a more “natural”setting they need not be restricted to a clinic setting. Measurements Xneed not necessarily consist of the same interview questions as inmeasurements U. When available, the supervised learning process can takeadvantage of measurements X and assessments Z in order to provideimproved estimates of Y. In such a manner, the system allows forevaluation and comparison of any known, expertly administered testsequence (assessments Y in FIG. 4), and any automated interview andanalysis system. Therefore, in accordance with this embodiment, humanexperts score recorded video interviews (based on standard scales,assessment Z). In parallel, the following may be extracted while theautomated interview is being conducted, or in accordance with post-hocanalysis of recorded video. Extracted items may include, for example,one or more of:

-   -   Visual: action units, head pose, gestures, personal hygiene,        etc.    -   Prosodic (speech tone): utterance duration, jitter, ratio of        time speaking, pitch range, etc.    -   Natural language: histogram of keyword frequencies in        transcribed text, “bag-of-words” etc.        It is thereafter possible to utilize one or more of supervised        on unsupervised learning techniques to determine one or more        correlations between disease progression, human scoring        indicative of disease (for example, assessments Y in FIG. 4),        and one or more extracted items. That is, the computer system        can learn on various data inputs, including the answers to        questions and final score of the expertly-administered scales,        and the one or more extracted items or scores available from the        automatically-administered system in order to determine        progression of disease. In other words, the        expertly-administered scales are designed to track disease        progression. By correlating the automated responses to the        expertly-administered scales, the system can learn to determine        progression of disease based on the automated responses of the        user.

Analysis of such presented interviews and scales may provide furtherpossibilities to extract data. As noted above, collected data may befurther analyzed to determine correlations between data collected andrisk of patient progression of disease, or risk of one or moreadditional negative actions to be performed by the patient. Thus,observation of actions of a patient may be determined to be indicativeof the likelihood of disease progression in the future, based uponobserved progression from other patients in the past. For example, if anaverage patient shows that progression of a tremor for small to mediumover a week, this may be indicative of a fast progression, while if thissame progression takes two months, this may be indicative of a slowprogression. Based upon this determination, a measured progression on aparticular patient may then give insight into how quickly the patientmay then progress on to a large tremor. Similarly, such a measuredprogression of tremor may also be indicative of a next symptom to beinitiated. So, for example, tremors reaching a particular level may alsoindicate the initiation of changes in vision. The automated systemdescribed in this application aims to provide an automated system foranalyzing patient responses, and to then measure disease progression,and predict likely future behavior.

Outcomes of correlation may be then used to determine risk profiles ofpatients, and may further be used to determine when a patient should becontacted or otherwise the subject of an intervention. This interventionmay be performed by an alerted human, or may include a further automatedsystem for intervening with a patient, and may present an appropriateadditional interview based upon the current state and risk profile ofthe patient. These additional automated interview systems may includeincreased observation (or decreased) based upon one or more sensor typesin accordance with estimates of disease. Thus, certain diseasedeterminations may justify higher resolution monitoring (i.e. morefrequently, or higher resolution video, depending on what is to bemonitored. For example, motion, movement, emotion, etc. may be monitoredmore frequently (perhaps continuously) while capture of audio videoresponses to interview questions may be performed less frequently.

To make an automated interview appear as close to an in-person interviewas possible, a selected neutral image with added animated expression canbe used. A user's response or motion, as well as the interviewquestions, can be used to determine the animated expression. Forexample, as shown in FIG. 5, if a user says “I'm happy my team won thegame!”, a computer algorithm using visual, voice, and language analysisdetermines that the user is happy, then instructs a smiling expressionto be added to the neutral image. When a user changes his/her positionwith respect to camera, the neutral face can also adjust its gaze tofollow the user. The neutral image can be from, for instance, apreferred doctor, caregiver, family member, friends, celebrity, acartoon character, or even abstract images. This approach can also beused for automated intervention systems.

Finally, once determinations of disease progression may be determinedbased upon the automated interviews, and accompanying analysis, actualpatient outcomes (i.e. actual known progression of disease, comprisesyet another data input (feedback system) that may be provided to thesupervised or unsupervised learning system to allow for further trainingand modification of the system to better predict disease in the future.

Furthermore to the extent any such relationship between a measuredcharacteristic and disease has not yet been defined, in accordance withan alternative embodiment of the disclosure, collected data may beprocessed to determine any such relationships to allow for use in thefuture. Different demographic groups may be employed to determinecharacteristics in these particular demographic groups, and thus allowfor targeted diagnosis and monitoring of disease. The use of supervisedor unsupervised learning techniques may be employed to analyze such datato determine any applicable relationships.

It should be noted that any of the above-noted embodiments of thedisclosure may be provided in combination or individually. Furthermore,the system may be employed in mobile devices, computing devices, cloudbased storage and processing. Camera images may be acquired by anassociated camera, or an independent camera situated at a remotelocation. Processing may be similarly be provided locally on a mobiledevice, or a remotely at a cloud-based location, or other remotelocation. Additionally, such processing and storage locations may besituated at a similar location, or at remote locations.

Embodiments of the subject matter and the functional operations, forexample, the remote information capture apparatus 1000 and remote dataand computing device 3000 in FIGS. 1-2 and the processes described indetail above with reference to FIGS. 3-5, can be implemented in digitalelectronic circuitry, in tangibly-embodied computer software orfirmware, in computer hardware, including the structures disclosed inthis specification and their structural equivalents, or in combinationsof one or more of them. Embodiments of the subject matter described inthis specification can be implemented as one or more computer programs,i.e., one or more modules of computer program instructions encoded on atangible non-transitory program carrier for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can also beor further include special purpose logic circuitry, e.g., an FPGA (fieldprogrammable gate array) or an ASIC (application-specific integratedcircuit). The apparatus can optionally include, in addition to hardware,code that creates an execution environment for computer programs, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program, which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code, can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub-programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a data communicationnetwork.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Computers suitable for the execution of a computer program include, byway of example, can be based on general or special purposemicroprocessors or both, or any other kind of central processing unit.Generally, a central processing unit will receive instructions and datafrom a read-only memory or a random access memory or both. The essentialelements of a computer are a central processing unit for performing orexecuting instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the user device, which acts as aclient. Data generated at the user device, e.g., a result of the userinteraction, can be received from the user device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of any subjectmatter described in this disclosure or on the scope of what may beclaimed, but rather as descriptions of features that may be specific toparticular embodiments of the subject matter described in thisdisclosure. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

1. A system comprising one or more computers in one or more locationsand one or more storage devices storing instructions that, when executedby one or more computers, cause the one or more computers to performoperations comprising: providing, to a user interface of a user device,a sequence of questions for a user to respond to in an interactivemanner, wherein each of the questions following the first questions isadaptive based on the user's response to one or more of the previousquestions in the sequence; capturing motion and appearance of the userin a video sequence while the user is responding to the sequence ofquestions; and analyzing the motion of the user in the video sequence todetermine one or more indications of a disease or of a change in adisease progression.
 2. The system of claim 1, wherein capturing motionof user comprises capturing one or more of the following: body motion,eye motion, emotions, heart rate, breathing patterns, skin tone, orspeech tone of the user.
 3. The system of claim 1, wherein theoperations comprising: based on responses of the user to the sequence ofquestions, selectively requesting the user to perform one or moreactions to complete a medical test.
 4. The system of claim 3, whereincapturing motion and appearance of the user in a video sequence furthercomprises: capturing motion and appearance of the user while the user isperforming the one or more actions to complete a medical test; andwherein analyzing the motion of the user in the video sequence comprisesanalyzing the motion and appearance of the user while the user isresponding the sequence of questions and while the user is performingthe one or more actions to complete the medical test.
 5. The system ofclaim 3, wherein requesting the user to perform one or more actionscomprises asking the user to hold the user device such that a camera ofthe user device focuses on a static object.
 6. The system of claim 5,wherein capturing motion and appearance of the user comprises capturingcamera motion of the user device with respect the static object.
 7. Thesystem of claim 6, wherein analyzing the motion of the user in the videosequence to determine one or more indications of a disease or of achange in a disease progression comprises: measuring the user's handstability or tremor based on the captured camera motion; and determiningone or more indications of a disease or of a change in a diseaseprogression based on the measured hand stability or tremor.
 8. Thesystem of claim 3, wherein requesting the user to perform one or moreactions comprises asking the user to look at a moving object displayedon the user interface.
 9. The system of claim 8, wherein capturingmotion and appearance of the user comprise capturing eye movement of theuser while the user is looking at the moving object.
 10. The system ofclaim 9, wherein analyzing the motion of the user comprises: measuringthe user's ability to focus on the moving object; and determining one ormore indications of a disease or of a change in a disease progressionbased on the user's ability to focus on the moving object.
 11. Acomputer-implemented method comprising: providing, to a user interfaceof a user device, a sequence of questions for a user to respond to in aninteractive manner, wherein each of the questions following the firstquestions is adaptive based on the user's response to one or more of theprevious questions in the sequence; capturing motion and appearance ofthe user in a video sequence while the user is responding to thesequence of questions; and analyzing the motion of the user in the videosequence to determine one or more indications of a disease or of achange in a disease progression.
 12. The method of claim 11, whereincapturing motion of user comprises capturing one or more of thefollowing: body motion, eye motion, emotions, heart rate, breathingpatterns, skin tone, or speech tone of the user.
 13. The method of claim11, further comprising: based on responses of the user to the sequenceof questions, selectively requesting the user to perform one or moreactions to complete a medical test.
 14. The method of claim 11, whereincapturing motion and appearance of the user in a video sequence furthercomprises: capturing motion and appearance of the user while the user isperforming the one or more actions to complete a medical test; andwherein analyzing the motion of the user in the video sequence comprisesanalyzing the motion and appearance of the user while the user isresponding the sequence of questions and while the user is performingthe one or more actions to complete the medical test.
 15. The method ofclaim 14, wherein selectively requesting the user to perform one or moreactions comprises asking the user to hold the user device such that acamera of the user device focuses on a static object.
 16. The method ofclaim 15, wherein capturing motion and appearance of the user comprisescapturing camera motion of the user device with respect the staticobject.
 17. The method of claim 16, wherein analyzing the motion of theuser in the video sequence to determine one or more indications of adisease or of a change in a disease progression comprises: measuring theuser's hand stability or tremor based on the captured camera motion; anddetermining one or more indications of a disease or of a change in adisease progression based on the measured hand stability or tremor. 18.The method of claim 13, wherein selectively requesting the user toperform one or more actions comprises asking the user to look at amoving object displayed on the user interface.
 19. The method of claim18, wherein capturing motion and appearance of the user comprisecapturing eye movement of the user while the user is looking at themoving object.
 20. The method of claim 19, wherein analyzing the motionof the user comprises: measuring the user's ability to focus on themoving object; and determining one or more indications of a disease orof a change in a disease progression based on the user's ability tofocus on the moving object.
 21. One or more non-transitorycomputer-readable storage media storing instructions that, when executedby one or more computers, cause the one or more computers to performoperations comprising: providing, to a user interface of a user device,a sequence of questions for a user to respond to in an interactivemanner, wherein each of the questions following the first questions isadaptive based on the user's response to one or more of the previousquestions in the sequence; capturing motion and appearance of the userin a video sequence while the user is responding to the sequence ofquestions; and analyzing the motion of the user in the video sequence todetermine one or more indications of a disease or of a change in adisease progression.